Systems and methods of capturing large area images in detail including cascaded cameras and/or calibration features

ABSTRACT

Systems and methods are disclosed relating to acquisition of images regarding large area objects or large areas. In one exemplary embodiment, there is provided a method of obtaining or capturing, via a first system that includes one or more first image capturing devices, overview images, wherein the overview images depict first areas, as well as obtaining or capturing, via a second system that includes a plurality of image capturing devices, detail images characterized as being related to each other along an image axis. Moreover, the detail images may depict second areas that are subsets of the first areas, they may be arranged in strips parallel to the image axis, and they may have a higher resolution than corresponding portions of the first images.

BACKGROUND

1. Field

The present invention relates to photogrammetry, and, more particularly,to systems and methods consistent with arrays of image capturing devicesdirected to the acquisition of images regarding large area objects orlarge areas.

2. Description of Related Information

Aerial and satellite imagery of the earth is used for a wide range ofmilitary, commercial and consumer applications. Recent innovationssometime include components that process and compress large amounts ofimages or serve entire photo imagery maps on the Internet, and advancessuch as these have further increased the demand for imagery. However,existing systems often involve overly complex components, require highcapital expenditures, and/or have high operating costs, among otherdrawbacks. They are unable to yield imagery within narrower timeframesand operating regimes, or otherwise provide the higher resolution,presently desired.

In general, existing photogrammetry imagery solutions are failing tomeet the increasing demand for more timely and higher resolutionimagery. According to principles consistent with certain aspects relatedto the innovations herein, camera systems used for aerial photogrammetrymust address two conflicting requirements.

First, it is vital that the camera system's lens and focal systemparameters (known as Interior orientation), as well as its position inspace and look angle (known as Exterior orientation) are preciselycalculated. A photogrammetric solution known as bundle adjustment may beused to calculate Interior and Exterior orientation information for thecamera and for each photo taken by the camera. Such calculations oftenrepresent a pre-requirement for enabling merging of individual photosinto seamless photomaps. One way of achieving the required level ofaccuracy is to take multiple photos, with a large amount of redundantoverlap between photos. Common features visible in multiple photos canthen be identified and used to calculate camera interior and exteriorparameters.

Second, it is desirable that aerial surveys be completed quickly. Thisprovides several advantages like reduced operating costs and minimizeddelays stemming from unfavorable environmental or surveying conditionssuch as inclement weather. An effective way to increasing the amount ofground area captured, measured in km² per hour, is to minimize theamount of redundancy between photos.

As such, the need to increase redundancy between photos to enableaccurate photogrammetric positioning of the photos must be balanced withthe need to decrease redundancy between photos to complete surveys at alower cost.

One existing approach uses “push-broom” linear detector arrays tominimize redundant capture and maximize capture rate. This approachminimizes the amount of redundancy and so increases capture rate.However, one drawback of this approach is that it sacrifices positionalaccuracy calculated from redundancy in the photos themselves, and soother complex methods must be used to accurately calculate camera systemposition information.

Another existing approach is to increase the size of the camera systembeing used, i.e., in terms of the megapixel count for the cameras orcamera arrays. Here, for example, multiple sensors and/or lenses may becombined in a single unit to maximize the megapixel count for the camerasystem. While this approach may increase the megapixel size of thecamera system, it fails to address reduction of redundancy betweenphotos.

Various systems are directed to minimizing amounts of redundant overlapbetween photos in a survey. Some existing camera systems, for example,are mounted in a fully gyroscopically stabilized platform which in turnis mounted in an aircraft. These systems may insulate the camera(s) fromexcessive yaw, pitch and/or roll, and enable a lesser amount ofredundancy to be used between photos. However, such stabilizationsystems are expensive and heavy, and suffer from drawbacks like highercamera system costs and the need for larger aircraft to fly the survey.

Other existing systems adapted to estimating camera pose and reducingredundant photo overlap requirements sometimes include one or more IMU(Inertial Measurement Unit) systems with the camera system to providemeasurement of the camera's yaw, pitch and roll. Such IMU systems,however, are complex and expensive, and the ability to utilize units ofsufficient accuracy is often constrained by export restrictions thatprohibit their use in many countries.

Certain other existing systems may include D-GPS (Differential GPS)units that enable estimation of the camera systems position when eachphoto is taken. These units, with appropriate post-survey (i.e.,post-flight) processing, allow position to be estimated to centimeteraccuracy. However, D-GPS units are expensive, and typically require adirect signal path to the GPS satellites in order to measure the signalphase later used to calculate precise position. Thus drawbacks of thesesystems include the requirement that aircraft must take verygradual/flat turns at the end of each flight line in a survey, to ensurethat portions of the aircraft such as a wing do not block the D-GPSantennae's view of the satellites. These gradual/flat turns addsignificantly to the amount of time required to fly a survey.

Still other existing systems provide improved photogrammetric solutionaccuracy via use of industrial grade high quality lenses, which canminimize the amount of Interior orientation error induced by lensdistortions. However, such high quality lenses add significantly to thecost of the camera system.

Even with such techniques, aerial surveys still require a significantamount of overlap between photos in order to ensure production of highquality photomaps. The amount of overlap between photos varies dependingon the application and desired quality. A common overlap is 30/80,meaning 30% side overlap with photos in adjacent parallel flight lines,and 80% forward overlap with photos along a flight line. This amount ofoverlap allows a feature to be identified on average in about 5 photos,which, in combination with the stability and position techniquesdiscussed above, is sufficient to enable accurate photogrammetric bundleadjustment of photos.

However, side overlap of 30% and forward overlap of 80% means that only14% of each photo covers new ground. About 86% of the photo informationtaken is redundant in terms of the final photomap product produces, soaerial surveys are fairly inefficient in terms of the amount of flyingrequired to cover an area. Also, the redundant photo data must be storedand later processed, which further increases costs.

While greater levels of redundancy, or overlap, increase the ability toprecisely calculate Exterior and Interior orientation for the camerasystem, such redundancy is largely wasted when creating a finalphotomap. This is because significantly more redundant imagery iscaptured than needed to create a photomap, which also increases the timeand cost required to fly a survey. A satisfactory balance between theseconsiderations is not available in a variety of other known systems,which all suffer from additional shortcomings.

For example, many existing systems for aerial photography require veryexpensive camera solutions that are typically purpose-built for theapplication. Such systems suffer the drawback that they cannot use COTS(Commercial Off the Shelf) cameras/hardware. Further, the heavy weightand high cost of these camera systems often requires the use oftwin-engine turbo-prop aircraft, which further drives up operating costssince these aircraft are much more expensive to operate than commonsingle engine commercial aircraft like the Cessna 210. Moreover,specific mounting requirements for such camera systems frequentlyrequire custom modification of the aircraft in order to mount the camerasystem.

Further, conventional large format aerial survey cameras are typicallylarge, heavy and expensive. It is often impossible to configure systemsof such cameras to take oblique photos at the same time as taking nadirphotos. Oblique photography is very widely used in intelligencegathering and military applications, and has recently become popular forconsumer applications. Oblique photomaps provide a view of objects suchas houses from the side, where as nadir, or overhead, photomaps lookfrom directly overhead and don't show the sides of objects. Obliquephotography is also desirable to enable textures to be placed over 3Dobject models to increase realism. Existing systems that do provideoblique imagery often suffer additional limitations. For example,capture rates can be very low, and the aircraft typically must fly atlow altitudes in order to capture high resolution oblique images.Moreover, minimal overlap is generally provided between photos fromdifferent obliques, making it difficult or impossible to createphotogrammetrically accurate photomaps.

Furthermore, many existing systems have limited resolution (megapixels)per image and use much of their available resolution to captureredundant data used to accurately calculate camera position and pose.These systems suffer drawbacks when identification of smaller objectsfrom the images is desired, such as the requirement that they flysurveys closer to the ground to capture images of high enough resolutionto identify such objects. For example, a camera system must survey (fly)at 3,000 feet altitude to capture 7.5 cm pixel resolution photos using aVexcel UltraCam-D camera. Flying at such a low altitude causes multipleproblems. First, turbulence and thermals are much worse at these loweraltitudes, which makes the flying rougher and more difficult for thepilot, and decreases the stability of the camera system. Secondly,flights over urban areas are more difficult at these altitudes, as ATC(Air Traffic Control) has to juggle the flight paths for the surveyaircraft—which needs to fly a consistent set of flight lines—withincoming and outgoing flights from airports surrounding the urban area.Interruptions in survey flights at these altitudes cause significantdelays in capturing the survey, further increasing costs.

Many existing systems also require large amounts of data storage onboardthe platform or aircraft. These systems typically include local imagecapturing systems and/or storage devices, to which image data istransmitted or downloaded from the cameras. Often, the storage must beboth fast enough to store photo data streaming from the cameras, andcapable of storing enough data to enable a reasonable amount of flyingtime. Further, many such systems use RAID based hard disk storagesystems to store in-flight captured data. However, hard disks aresensitive to low air pressure at higher altitudes, which can result inhead crashes or other data losses or errors.

In sum, there is a need for systems and methods that may adequatelycapture and/or process large area images in detail by, for example,utilization of one or more camera systems or arrays having imagecapturing/processing configurations that provide features such specifiedoverlap characteristics, the ability to create detail photomaps, amongothers.

SUMMARY

Systems and methods consistent with the invention are directed to arraysof image capturing devices, and processes associated therewith, thatacquire/process images of large area objects or large areas.

In one exemplary embodiment, there is provided a method of capturing,via a first system that includes one or more first image capturingdevices, overview images, wherein the overview images depict firstareas, and capturing, via a second system that includes a plurality ofimage capturing devices, detail images characterized as being related toeach other along an image axis. In one or more further embodiments, thedetail images depict second areas that are subsets of the first areas,are arranged in strips parallel to the image axis, and have a higherresolution than corresponding portions of the first images.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the invention, as described. Further featuresand/or variations may be provided in addition to those set forth herein.For example, the present invention may be directed to variouscombinations and subcombinations of the disclosed features and/orcombinations and subcombinations of several further features disclosedbelow in the detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which constitute a part of thisspecification, illustrate various embodiments and aspects of the presentinvention and, together with the description, explain the principles ofthe invention. In the drawings:

FIG. 1 is a block diagram of an exemplary system consistent with certainaspects related to the innovations herein.

FIGS. 2A-2B are block diagrams of exemplary systems consistent withcertain aspects related to the innovations herein.

FIG. 3 is a block diagram of an exemplary system consistent with certainaspects related to the innovations herein.

FIG. 4 is a diagram of an exemplary system including overview and detailimage capturing devices consistent with certain aspects related to theinnovations herein.

FIG. 5A illustrates one exemplary implementation including an externalpod mounted on a small single engine aircraft consistent with certainaspects related to the innovations herein.

FIG. 5B illustrates one exemplary implementation of an image capturingsystem consistent with certain aspects related to the innovationsherein.

FIGS. 6A-6B are diagrams illustrating exemplary overview and detailimage representations consistent with certain aspects related to theinnovations herein.

FIGS. 7A-7B are diagrams illustrating further exemplary overview anddetail image representations consistent with certain aspects related tothe innovations herein.

FIGS. 8A-8B are diagrams illustrating image representations havingtypical overlap conditions.

DETAILED DESCRIPTION OF SOME EXEMPLARY EMBODIMENTS

Reference will now be made in detail to the invention, examples of whichare illustrated in the accompanying drawings. The implementations setforth in the following description do not represent all implementationsconsistent with the innovations claimed herein. Instead, they are merelysome examples consistent with certain aspects related to the invention.Wherever possible, the same reference numbers will be used throughoutthe drawings to refer to the same or like parts.

Many systems and image capturing devices are used in terrestrial,airborne and space-borne platforms to acquire images of large areaobjects or large areas. These systems and platforms can be implementedwith a variety of components, including cameras, processing components,data stores, telescopes, lenses or other devices having specializedcomponents for capturing and/or processing images.

FIG. 1 illustrates a block diagram of an exemplary system 100 consistentwith certain aspects related to the present invention. Referring to FIG.1, system 100 may comprise a first system 110 that acquires overviewimages 112, and a second system 120 that acquires detail images 122,124, 126. According to some embodiments, the overview images may becharacterized by a first or overview axis 114. Similarly, the detailimages 122, 124, 126 may be arranged in strips along a second or detailaxis 130. Further, the first and second systems 110, 120 may eachinclude one or more image capturing devices, for example, cameras(throughout this disclosure, the broad term “image capturing device” isoften referred to as “camera” for purpose of convenience, notlimitation). As set forth in more detail below, innovations consistentwith the arrangements herein provide systems and methods having numerousadvantages, including the ability to accurately capture high resolutiondigital images over large areas for airborne or space-borne photomapsurveys at a much faster rate and shorter survey flight time thanexisting systems.

Further, according to some aspects of the innovations herein, first andsecond systems 110, 120 may include arrays of digital image capturingdevices, such as cascaded groups of multiple cameras mounted in rigid orsemi-rigid mounts. Persons of ordinary skill in the art will appreciatethat such mounting details are exemplary. For instance, rigid orsemi-rigid mounting system can describe any apparatus capable ofaccurately defining relative position of the multiple and cascadedgroups of cameras. Such a mounting system might be embodied via avariety of permutations, for example, it might comprise a physical rigidstructure, such as mounting the cameras into a pod enclosure, it mightcomprise cameras keeping independent but accurate station relative toone another, such as cameras mounted in multiple distinct aerial orsatellite systems with a local referencing system to define relativecamera positioning between satellites, etc.

System 100 of FIG. 1 is also exemplary with regard to variousconfigurations that may be present between or among systems 110, 120and/or their image capturing devices. For example, FIGS. 2A-2B are blockdiagrams illustrating differing arrangements of the first system 110 andthe second system 120 consistent with aspects related to the innovationsherein. FIG. 2A shows an implementation wherein the first system 110 andthe second system 120 are located in one fixed location, such as on anaerial platform, in a satellite, etc. FIG. 2B shows anotherimplementation wherein the innovations reside in just one of system,specifically, here, in the second system 120. In this exemplaryimplementation, innovations consistent with acquisition/processing ofparticular images may occur primarily via the second system 120. Here,relationship information between the first system 110 and the secondsystem 120, among arrays of cameras located therein, or among imagesobtained therefrom, is typically known or determinable, however, theinnovations described herein are resident on or associated primarilywith the second system 120. This arrangement may be useful, for example,when certain images, such as overview images, are obtained from a thirdparty provider, while the remaining images are obtained via the secondsystem 120. Lastly, while FIG. 2B illustrates the innovations residingin the second system 120, a similar arrangement may also exist withrespect to the first system. As also depicted for purpose ofillustration in FIGS. 2A-2B, the first system 110 may include one ormore first image capturing devices or cameras 210A and the second system120 may include one or more second image capturing devices or cameras220A.

Such exemplary camera arrays may be configured such that one or morecameras capture photos with very high amounts of overlap, e.g., to helpfacilitate accurate calculation of camera system Interior and Exteriororientation. Further, a second cascaded sub-groups of cameras may bearranged to capture images with minimal overlap but high detail, e.g.,to help facilitate processes such as refining the photogrammetricInterior and Exterior orientation, providing the photo image data neededto create detail photomap surveys, etc. Persons of ordinary skill in theart will appreciate that such delineations are exemplary, andconfigurations of cascaded cameras can be changed or tuned to specificapplications. For example, cameras used to capture high-redundancyphotos for calculating Exterior and Interior orientation can also beused to create lower-resolution overview photomaps for the survey.Further, cameras used for capturing low-redundancy high detail photosused to create detailed photomaps may also be used to refine Exteriorand Interior estimates for the camera system.

In certain implementations, some cameras may be configured to maximizethe amount of redundancy and overlap between photos, or otherwise enablemore precise calculations of the Interior and Exterior orientationsrelated to the camera systems. In further implementations, other camerasmay be arranged to minimize the amount of redundancy and overlap betweenphotos, or otherwise configured to enable creation of final detailphotomap surveys with a minimum amount of wasted redundant photoimagery.

FIG. 3 is a block diagram of another exemplary system consistent withcertain aspects related to the innovations herein. As shown in FIG. 3, aunitary platform or module 310 may include or embody both the firstsystem 110 and the second system 120. According to furtherimplementations, the platform 310 may also various arrangements and/orarrays of first and second image capturing devices or cameras 210A,210A′, 220A, 220A′, etc. Such arrangements and arrays of cameras may beconfigured to provide the various types of images described herein. Oneexemplary implementation of such an arrangement is set forth in moredetail in connection with FIG. 4, below. Advantages of implementationsconsistent with these arrangements include the ability to mount thesystems in an external camera pod, enabling use of the camera systems instandard aircraft without custom modifications, as well as reducedweight and size for the camera system, enabling it to be used in a“small” aircraft (e.g., a single engine aircraft of lesser expense, suchas a Cessna 210 or Diamond DA42 Twin Star), and also to enable it to beused in UAV (Unmanned Airborne Vehicle) aircraft.

Aspects of the innovations herein are also directed to overlap featuresexisting between the cameras, the images, or both, as well asinterrelationship of several such overlap features. In oneimplementation, with respect to overview images captured by the firstsystem, exemplary cameras may be configured with wide-angle lenses andused to capture photos with a very large amount of overlap. Photoscaptured by these cameras cover a larger area per photo. This very highamount of overlap redundancy results in ground points being visible inmany more photos than prior art camera systems, enabling precisepositioning of Interior and Exterior orientation even without the use ofa stabilised platform. For example, overlap of such overview images maybe characterized in the range of 45-65/94-99 (45%-65% side overlap and94%-99% forward overlap with regard to an axis), or narrower.Specifically, captured overview images may have side overlap redundancyof between about 45% to about 65% with images that are laterallyadjacent the first axis, as well as forward overlap redundancy betweenabout 94% to about 99% with images that are longitudinally adjacent thefirst axis. Narrower ranges include between about 50% to about 60% sideoverlap and between about 95% to about 99% forward overlap, betweenabout 98% and about 99% forward overlap, about 50% side overlap andabout 99% forward overlap, among others consistent with the parametersset forth herein. According to additional expressions of overlapconsistent with the innovations herein, overview images may also becaptured such that the images have overlap redundancy characterized inthat a same imaged point is captured: in a quantity of overview imagesgreater than about 30 and less than about 100, in an average of about 40to about 60 images, in an average of about 50 images, or in a maximum ofabout 100 images, depending upon the systems and processes involved. Afurther expression of overlap may also include characterization in thata same imaged point is captured in a quantity of about 500 images, asexplained in connection with FIG. 7A, below.

Further aspects of the innovations herein may also include arrays of oneor more cameras configured with longer focal length lenses and are usedto capture detail imagery to generate the detailed photomaps for thesurvey. Low amounts of overlap on these cameras may minimize redundancyand so maximize use of the photo imagery for the detail survey, and mayprovide other advantages such as significantly reducing the overallcosts and time required to complete a survey. Here, for example, onemeasure of overlap of such detail images consistent with the innovationsherein is characterized by a photo view overlap among the second imagecapturing devices is between about 0% and about 10%.

FIG. 4 is a diagram of an exemplary system including overview cameraarrays and detail camera arrays consistent with certain aspects relatedto the innovations herein.

Referring to FIG. 4, a unitary module 400 is disclosed including aplurality of overview cameras 410, at least one data store 430/430A, afirst array of detail cameras 420A, a second array of detail cameras420B, a third array of detail cameras 420C, and a fourth array of detailcameras 420D, etc. These arrays of detail cameras may be used, forexample, to obtain images of the various views set forth below at thesame time while flying a single survey, such as multiple oblique views,overhead nadir views, etc. One of ordinary skill in the art wouldrecognize that the quantities (i.e., of both the cameras and of thearrays) of detail cameras may be adjusted according to thespecifications known to an ordinary artisan to provide for image resultsdesired. Advantages consistent with such implementations include theability to configure and/or reconfigure a module 400 to target differentsurvey requirements, such as nadir photo maps, oblique photo maps,infrared photomaps, or any combination of these or other requirementsthat may arise. Further, innovations consistent with modules like thatof FIG. 4 provide for improved initial estimates of the look angle forthe Detail cameras relative to the Overview cameras.

Further, implementations consistent with FIG. 4 allow for the use oflow-cost COTS (Commercial Off The Shelf) cameras, rather than requiringindustrial quality and expensive camera systems as do many existingsystems. According to some aspects of the innovations herein, systemsand methods may include image capturing devices that areremovably/modularly mounted in a platform such that individual imagecapturing devices are replaceable. For example, one or both of the firstsystem or the second system may be configured/designed with removablemounting systems such that the image capturing devices may beinterchanged with different image capturing devices. Exemplary imagecapturing devices, here, may include COTS cameras installed such thatthey may be individually removed for repair, replacement and/or upgrade.This provides particular innovations, such as the ability to quicklytake advantage of new advances in digital photography, like rapiddevelopments in and the low cost of next generation professional D-SLR(Digital Single Lens) cameras. Use of such cameras has advantages suchas reducing the cost of the camera system in total, and also enablesready and rapid upgrade as new D-SLR cameras are released with increasedresolution, higher performance, and/or lower cost.

As shown in FIG. 4, platforms or modules 400 consistent with theinvention may also include a data store 430 or a plurality of suchcomponents, one associated with each camera 430A. With regard to thelatter, some of the innovations herein include features of compressingand/or storing images in association with each camera, rather thanrequiring captured photos to be stored in a central storage system,transmitted, etc. Further, features directed to parallel compression andstorage of photos on each camera increases the maximum throughput andstorage for the camera system, which allows surveys to be flown at afaster rate, enabling more data to be stored and flight time to beincreased.

Such parallel compression and storage on each camera also increasesstorage reliability, as it allows use of Compact Flash or othersolid-state media on each camera. Existing systems typically store theraw linear sensor as 12 to 16 bit data stored to a central storagesystem. In contrast, by performing compression on each camera inparallel, innovations herein allow data to be converted to a gammacolour space such as YCbCr. This allows data to be stored as 8 bit datasince increased bit depth is typically only needed for raw linear data,and further allows compression of images prior to storage on eachcamera's data store. Conversion to a gamma color space and compressioncan enable about a 10-fold reduction in storage space requirements. Forexample, in system having 14 cameras each with its own 32 GB CompactFlash memory card, the total of 448 GB of storage can be equivalent toupwards of about 4,500 GB or 4.5TB of storage of raw uncompressed photodata. Further advantages relate to features of parallel operation andavoiding transmissions of image data or any other signals from thecameras to the flight control computer system, such as increasingcapture rate for the camera system, reducing post-processingrequirements, increasing robustness by reducing cabling and signallingrequirements, among others.

Systems consistent with the exemplary implementations of FIGS. 1-4 maybe utilized to implement image capturing methodologies consistent withcertain aspects related to the innovations herein. These systems mayinclude the image capturing devices from which the images describedherein are obtained or captured, as well as other elements that processand store such images. According to some processes performed by thesesystem, exemplary methods may include obtaining or capturing overviewimages, wherein the overview images depict first areas, as well asobtaining or capturing detail images characterized as being related toeach other along an image axis. Here, the overview images may beobtained or captured via a first system or array including first imagecapturing devices. Further, the detail images may be obtained orcaptured via a second system or array including second image capturingdevices. Moreover, the detail images captured may depict second areasthat are subsets of the first areas, they may be arranged in stripsparallel to the image axis, and/or they may have a higher resolutionthan corresponding portions of the first images.

With regard to the detail images, some of the image capturing processesherein are directed to capturing detail images at a resolutionsufficient to produce a detail photomap. Regarding the capture of thesedetail images and/or the detail images themselves, determiningsufficient resolution, here, is well known to those skilled in the art.Such determinations being consistent, for example, with those related toU.S. Pat. Nos. 6,078,701, 6,694,064, 6,928,194, 7,127,348, and7,215,364, and/or U.S. patent application publication Nos.2002/0163582A1, 2005/0265631A1, and 2007/0188610A1. Further, someaspects of the innovations herein are particularly well suited tocreation of detail photomaps of much higher resolution than comparablesystems, i.e., wherein the detail images are captured at a resolutionsufficient to produce a detail photomap having a ground-pixel resolutionof at least 10 cm. Innovations herein consistent with the above enableadvantages such as one or more of enabling high-resolution surveys to becaptured from higher altitudes, reducing impacts associated with AirTraffic Control restrictions, providing smoother flying conditions,and/or reducing pilot/operator workload.

With regard to the overview images, some of the image acquisitionprocesses herein are directed to capturing images having overlap betweenimages characterized in that a same image point is captured in aquantity of images sufficient to enable accurate bundle adjustment.Other image acquisition processes herein are directed to capturingimages having overlap between images characterized in that a samefeature is captured in a quantity of images as required by bundleadjustment. Further, the bundle adjustment solution may derived as afunction of both the overview images and the detail images.

Bundle adjustment (see, e.g., Wolf, Elements of Photogrammetry, 1983,and Manual of Photogrammetry, 3rd Edition, American Society ofPhotogrammetry, 1966) is a known mathematical manipulation used toprecisely calculate the position, known as exterior orientation, andcamera calibration, known as interior orientation for each photo takenfor a terrestrial, airborne or space-borne survey using camera systems.

The bundle adjustment referred to herein simultaneously refinesestimates for ground point positions and for each photo's exterior andinterior orientation. A ground point position is identified as a featurein each photo. A requirement for bundle adjustment is to maximize theaverage and maximum number of photos in which a ground point isidentified. If a ground point is identified in too few photos, then thesolution is not very rigid and suffers both from accuracy errors andfrom an increased risk of blunders, where incorrectly identified groundpoints have been used in the bundle solution. Bundle adjustment is alsocapable of refining photos that have different poses, for example imageshaving different oblique angles or oriented differently.

According to the innovations herein, use of cascaded cameras allows theinterior and exterior orientation of photos taken by the detail camerasbe further refined through bundle adjustment. Using known bundleadjustment techniques, this may be achieved by identifying ground pointsvisible in images captured by overview cameras and in images captured bydetail cameras. As the overview cameras provide very high redundancy andthus accuracy in the bundle adjustment process, this serves as the basisfor calculating accurate interior and exterior orientation for photostaken with detail cameras, despite the limited amount of redundancy andoverlap provided by detail cameras. Advantages related hereto includethe ability to enable self-calibration of camera interior orientationparameters, such as lenses focal length and distortions, allowing lowercost professional grade lenses to be used and affording automation ofthe photomap photogrammetry process.

Further aspects of the innovations herein allow for all or a pluralityof cameras in the camera system(s) to have their shutters triggered atthe same time or at nearly the same time. In this context, ‘at nearlythe same time’ refers to a period of about 100 milliseconds given stableplatform (i.e., flying, pitch, yaw, etc.) conditions. This providesfurther rigidity to the bundle adjustment solution, as the camera systemcan be modelled more accurately, for example, by using known bundleadjustment methods for multiple-camera interior and exterior orientationrefinement.

FIG. 5A illustrates one exemplary implementation including an externalpod mounted on a small single engine aircraft 510 consistent withcertain aspects related to the innovations herein. Referring now to FIG.5A, one specific embodiment of this invention is to mount the camerasfor the camera system into a pod or removable enclosure 520, making itpossible to use the camera system on a standard small aircraft 510 suchas the Cessna 210 without requiring modifications to the airframe. FIG.5B illustrates an exemplary implementation of an image capturing systemconsistent with certain aspects related to the innovations herein. Asshown in FIG. 5B, a pod or removable enclosure 520 may include aplurality of overview/detail cameras 410/420, which may be grouped orarranged, e.g., in arrays, as set forth herein. Implementations such asthese shown in FIGS. 5A and 5B provide high accuracy without requiring astabilized mounting platform, and also enable sufficient weight and sizereduction allowing the camera system to be mounted in a UAV.

FIGS. 6A-6B are diagrams illustrating exemplary overview and detailimage representations consistent with certain aspects related to theinnovations herein. FIG. 6A shows one exemplary representation whereinmultiple cameras are configured to maximize the amount of detail imagedata 610 obtained in the unique area through the use of multiple detailcameras, while at the same time ensuring significant overlap existsbetween overview images 612 to enable accurate bundle adjustment.

The representation of FIG. 6A may be achieved, for example, using oneoverview camera (see, e.g., representative images 612, 616, 620, 624thereof) to capture interior and exterior orientation, and a one cascadegroup of nine cameras organized to capture detailed strips 610, 614,618, 622 or sub-portions of each overview photo in very high detail. Asset forth above, aspects of the innovations herein may include rigid orsemi-rigid alignment of cameras in the camera system, which allowsphotos to be taken with minimal overlap between photos within the strip.Further, images may be taken often enough to ensure overlap existsbetween consecutive photos along a flight line, and flight lines may beorganized to ensure that there is overlap between strips of photos takenalong adjacent flight lines. Unlike existing systems where significantoverlap is required to perform accurate bundle adjustment, presentinnovations enable use of a minimal amount of overlap to exist betweensubsequent or adjacent photo strip details, which only needs to besufficient to later perform creation of a seamless photomap. As aresult, the redundancy required for a strip of photos from detailcameras is much less than with existing systems, which significantlydecreases survey time and costs.

Moreover, as many additional detail cameras as required may beconfigured in a cascaded fashion to capture detailed sub-portions of theoverview images for specific views, such as nadir overhead photomaps oroblique photomaps from different look angles. Because a single detailcamera may not have sufficient resolution to capture a sub-portion insufficient resolution for the desired survey, a group of detail camerasfor a specific view perspective may be organized in a strip to capture awider swath of the desired perspective. FIGS. 7A-7B are diagramsillustrating further exemplary overview and detail image representationsconsistent with certain aspects related to the innovations herein. FIG.7A shows three cascaded groups of detail cameras where the five cameras(see, e.g., images 730, 730A-E) provide a detailed vertical view, fourcameras (see, e.g., images 740) provide detailed left and right obliqueviews from alternating flight lines, and three cameras (see, e.g.,images 750, 750A-C) provide detailed front and back oblique views fromalternating flight lines. FIG. 7B illustrates a further embodiment,wherein multiple oblique views are provided by flying flight lines inalternating directions, for example, by obtaining four oblique viewsfrom two groups of oblique cameras.

The representation of FIG. 7A also illustrated another feature wheremultiple overview cameras 710, 720 are used, each oriented at adifferent pose. This exemplary feature increases the amount of overlapbetween photos considerably, allowing overlap between images that mightbe several flight lines away. As such, redundancy and rigidity of thefeature matching solution may be significantly increased between images.Further, combining multiple overview cameras with different posesenables the same ground point to be visible and measured on 500 or morephotos. This compares favorably with existing methods having 30%/80%overlap, which result in a ground point being captured in an average of5 photos.

Turning back to FIGS. 6B and 7B, these drawings illustrate the amount ofoverlap between images. Here, the overlap amounts to 50%/95% as thesetwo implementations are compared to that FIG. 8B, which shows the 30/80overlap as commonly used by prior art. Each of these figures show imagesor group of images in a sequence taken during a survey and adjacentimages or groups of images in the previous and next flight line for thesurvey. The large amount of redundancy allows bundle adjustment toaccurately refine photo interior and exterior position to sub-pixelaccuracy for the overview cameras.

FIG. 7A illustrates other exemplary features of the invention, such asusing two overview cameras 710, 720 to capture interior and exteriororientation, and three cascade groups of detail cameras 730, 740 and 750to capture an overview nadir detail view and two oblique detail views.When the aircraft survey is flown in alternating directions for eachflight line, then the two oblique views alternate direction, resultingin a total of four oblique detail views being captured in addition tothe overview detail view. Indeed, the ability to configure the camerasystem to specific survey mission requirements enables the simultaneouscapture of detail photomaps from different look angles, at the sametime. FIG. 7A, for example, allows production of a detail overheadphotomap and four detail oblique photomaps through a combination ofmultiple cascaded camera groups and the use of alternative flight lines.

Arranging strips of detail cameras into arrays or groups gives thecamera system a high virtual megapixel count. With respect to anexemplary system consistent with FIG. 7A, e.g., one implementation uses14 cameras, each being a 21 megapixel 35 mm D-SLR camera, yielding aneffective camera system resolution of several gigapixels in size. Inthis exemplary implementation, one overview camera 710 provides a nadiroverhead view connected to another overview camera 720 to provide a rearoblique overhead view. One cascade group of a plurality of detailcameras 730 may provide a detailed nadir survey imagery referencedwithin the first overview camera 710. Another cascade group of aplurality of detail cameras 750 may provide along track detailed obliquesurvey imagery referenced within the overview camera 720. Anothercascaded group of a plurality of detail cameras 740 may provide acrosstrack detailed oblique survey images which are referenced using a rigidcamera system body and/or referenced using overview camera imagery fromadjacent survey flight lines as shown in FIG. 7B.

FIGS. 8A-8B are diagrams illustrating image representations showingtypical overlap conditions of existing systems. FIGS. 8A-8B depictsrepresentations of an existing large format camera configuration, where810 is the total photo coverage of the ground for a single image and 820represents the portion that is unique to this photo when a survey isflown with a typical 30%/80% overlap. It can be seen that the uniqueportion of the photo is only a small percentage of the total photo area,and thus the remaining photo area is redundant in terms of the finalphotomap requirement.

Features herein associated with minimizing overlap between photoscaptured by detail cameras have advantages such as maximizing use ofimagery in resulting photomaps.

This allows surveys to be flown at higher altitudes and in less time.Flying surveys at higher altitude reduces impact on Air Traffic Controlin busy urban areas, and generally provides for smoother flyingconditions and lower pilot/operator work-load. Flying surveys in lesstime reduces the operating costs for the survey, and allows a survey toflown as soon as weather clears, rather than waiting for larger blocksof time with clear weather. Accordingly, innovations consistent with theabove may also greatly increase the likelihood of capturing a surveydespite inclement weather.

Further, aspects of the innovations herein that provides a high amountof overlap between photos captured by overview cameras enable“self-calibration”, or accurate modelling of interior orientation lensand sensor characteristics using existing bundle adjustmentself-calibration techniques. For example, as images are captured by thecascaded detail cameras are in turn mapped into the overview photos,such self-calibration modelling can be performed for detail cameras aswell as for overview cameras. Because innovations herein enable accurateself-calibration, low-cost COTS professional grade lenses can be used inthe camera system, instead of requiring the use of much more expensiveindustrial grade lenses.

Aspects of innovations herein also allow the use of IMU, D-GPS,stabilized platforms or other complex or expensive ancillary systems,which decreases the capital and operating cost for the camera system,and may reduce overall complexity. Still other advantages of theinnovations herein allow for an increase the accuracy of calculatedcamera position and pose, without the need for expensive D-GPS,stabilisation or IMU ancillary sub-systems.

As may be appreciated in connection with the strip of detail images 610in FIG. 6A, innovations herein also relate to the concept of the fieldof view for a group of detail cameras being very wide but very narrow.If the camera sensor platform pitches quickly enough in the direction ofthe narrow aspect of the field of view, it's possible that detail viewsof the ground might be missed. Although the risk of this happening ismitigated because the systems and methods herein may be practiced athigher (and, typically, much smoother) altitudes, the innovations mayalso include use of low-cost MEMS (Micro-Electro-Mechanical Systems)accelerometers to detect rapid changes in pitch. MEMS accelerometers arevery cheap (they are used in air-bags), though are not suitable for manyIMU measurements as they drift over time. However, implementations ofMEMS with platform or aircraft acceleration, rapid pitch, etc andrelated image capturing devices afford particularized advantage to thepresently described systems and methods. Innovations herein involve theuse of MEMS accelerometers to measure rapid short-term pitch, yaw orroll changes, and to use this information to increase the number ofshutter events and photos taken during these times of rapid change toensure that detail cameras with narrow fields of view still cover allthe required ground area even during rapid changes of sensor platformpose.

Finally, Digital Elevation Models (DEMs) are a common by-product fromthe photogrammetric bundle adjustment process. DEMs are useful in theirown right for applications such as flood and fire modelling, and arealso required to produce ortho-rectified photomaps using the usualprior-art methods as present in applications such as ER Mapper [Nixon,Earth Resource Mapping, www.ermapper.com]. The overall accuracy of DEMsis commonly much more important than the density of measurements for theDEM itself. Ortho-rectification commonly uses DEMs that are 1/10th orless resolution than the photo imagery being rectified. Aspects of theinnovations herein provide a high level of overlap between imagescaptured by overview cameras. A single ground point can typically beobserved in several orders of magnitude more photos than possible inexisting camera systems. As such, the redundancy of observations ofground points provided by innovations herein also enables production ofrobust and accurate DEMs.

It is to be understood that the foregoing description is intended toillustrate and not to limit the scope of the invention, which is definedby the scope of the appended claims. Other embodiments are within thescope of the following claims.

The invention claimed is:
 1. A method of capturing images comprising:capturing, via a first system that includes one or more first imagecapturing devices, overview images, wherein the overview images depictfirst areas; and capturing, via a second system that includes aplurality of second image capturing devices, detail images related toeach other along a detail image axis, wherein the detail images depictsecond areas that: are subsets of the first areas, are arranged instrips parallel to the detail image axis, and have a higher resolutionthan corresponding portions of the overview images, wherein aftercapturing the overview images and the detail images, a bundle adjustmentprocess to determine interior and exterior orientations of the detailimages is performed based on overview image overlaps and the detailimages, and detail photomaps are generated from the detail images usingthe interior and exterior orientations determined in the bundleadjustment process, and wherein the overview images have a redundancygreater than 17 images, an area of at least one overlapping overviewportion between a first and second overview image is greater than orequal to 50% of an area of one of the first and second overview images,and an area of at least one overlapping portion of a first and seconddetail image is less than or equal to 10% of an area of one of the firstand second detail images.
 2. The method of claim 1, further comprisingbundle adjusting an orientation of the first system, the second system,or one or more of the first and second image capturing devices usingdata associated with a spatial relationship among the overview imagesand the detail images.
 3. The method of claim 2 wherein a side overlapof the overview images is between 50% and 60% with images that arelaterally adjacent to an overview axis, and a forward overlap of theoverview images is between 94% and 99% with images that arelongitudinally adjacent to the overview axis.
 4. The method of claim 1,further comprising capturing images corresponding to multiple differentviews, including one or more oblique views and/or one or more nadirviews.
 5. The method of claim 1, wherein the detail photomaps have aground-pixel resolution of 10 cm or less.
 6. The method of claim 1,wherein the overview images are captured with an overlap between imagesto cause a same image point to be captured in a quantity of imagessufficient to enable accurate bundle adjustment.
 7. The method of claim1, wherein a digital elevation model is produced during the bundleadjustment process, the digital elevation model being used toortho-rectify the detail images into the detail photomaps.
 8. The methodof claim 1, further comprising: using the bundle adjustment process toself-calibrate an interior orientation of the one or more first imagecapturing devices and the plurality of second image capturing devices.9. A method of capturing images comprising: capturing, via a firstsystem that includes one or more first image capturing devices, overviewimages related to each other along an overview axis, wherein theoverview images depict first areas; and capturing, via a second systemthat includes a plurality of second image capturing devices, detailimages related to each other along a detail axis, wherein the detailimages depict second areas that are subsets of the first areas and thedetail images have a higher resolution than corresponding portions ofthe overview images; wherein the overview images captured have a sideoverlap between 50% and 60% with images that are laterally adjacent tothe overview axis, and a forward overlap between 94% and 99% with imagesthat are longitudinally adjacent to the overview axis, after capturingthe overview images and the detail images, a bundle adjustment processto determine interior and exterior orientations of the detail images isperformed based on the side and forward overview image overlaps and thedetail images, and detail photomaps are generated from the detail imagesusing the interior and exterior orientations determined in the bundleadjustment process, and wherein the overview images have a redundancygreater than 17 images, an area of at least one overlapping overviewportion between a first and second overview image is greater than orequal to 50% of an area of one of the first and second overview images,and an area of at least one overlapping portion of a first and seconddetail image is less than or equal to 10% of an area of one of the firstand second detail images.
 10. The method of claim 9, wherein imagescaptured correspond to multiple different views, including one or moreoblique views and/or one or more nadir views.
 11. An image capturingsystem comprising: a first system comprised of one or more first camerasand configured to capture overview images that are related to each otheralong an overview axis, wherein the overview images have forward overlapbetween 94% and 99% with images that are longitudinally adjacent to theoverview axis; and a second system comprised of a plurality of secondcameras and configured to capture detail images having a higherresolution than corresponding portions of the overview images, thedetail images related to each other along a detail axis, wherein thedetail images are captured such that overlap of regions of the detailimages is between 0% and 10%, wherein the one or more first cameras ofthe first system and second cameras in the second system are configuredsuch that interior orientation information and exterior orientationinformation for the detail images may be determined from a bundleadjustment of the detail images performed after the detail images andthe overview images are captured and based on overview image overlapsand the detail images, and detail photomaps are generated from thedetail images using the interior and exterior orientation information,and wherein the overview images have a redundancy greater than 17images, an area of at least one overlapping overview portion between afirst and second overview image is greater than or equal to 50% of anarea of one of the first and second overview images, and an area of atleast one overlapping portion of a first and second detail image is lessthan or equal to 10% of an area of one of the first and second detailimages.
 12. The system of claim 11, wherein interior orientation andexterior orientation is determined for images from one or more ofindividual cameras, groups of cameras, or a camera system as a whole.13. The system of claim 11, wherein the second system includes 3sub-arrays of second image capturing devices, wherein a first sub-arraycaptures detail nadir view images, a second sub-array captures firstoblique detail view images, and a third sub-array captures secondoblique detail view images.
 14. The system of claim 11, wherein theforward overlap of the overview images is between 98% and 99% withimages that are longitudinally adjacent to the overview axis.
 15. Thesystem of claim 11, wherein the second system is configured to captureimages corresponding to multiple different views, including one or moreoblique views and/or one or more nadir views.
 16. The system of claim11, wherein one or both of the first system or the second system areconfigured to include a removable mounting system such that the first orsecond cameras may be interchanged with different image capturingdevices.
 17. The system of claim 11, wherein multiple first cameras arearranged at different poses to maximize an amount of ground featureobservations included within multiple overview images taken by the firstcameras, wherein maximizing the amount of ground feature observationsprovides sufficient data to extract accurate photogrammetric bundleadjustment solutions.
 18. The system of claim 11, further comprising: aprocessing component that compresses image data local to each of thefirst and second cameras; and a solid state data store local to each ofthe first and second cameras and that is configured to store image datain parallel with regard to each of the first and second cameras.
 19. Thesystem of claim 11, wherein the second system includes between 5 and 13second cameras that capture strip-shaped sub-portions of the overviewimages.
 20. The image capturing system of claim 11, wherein the detailphotomaps have a ground-pixel resolution of 10 cm or less.
 21. The imagecapturing system of claim 11, wherein the overview images captured havea side overlap between 50% and 60% with images that are laterallyadjacent to the overview axis.
 22. An image capturing system comprising:a first system including one or more first image capturing devices thatcapture overview images, wherein the overview images depict first areas;and a second system including a plurality of second image capturingdevices that capture detail images related to each other along a detailimage axis; wherein the second image capturing devices are configured tocapture the detail images such that the detail images: depict secondareas that are subsets of the first areas, are arranged in stripsparallel to the detail image axis, and have a higher resolution thancorresponding portions of the overview images, wherein orientations ofthe detail images are determined through a bundle adjustment processthat is performed after the overview images and the detail images arecaptured, and that uses overview image overlaps, and detail photomapsare generated from the detail images using the interior and exteriororientations determined through the bundle adjustment process, andwherein the overview images have a redundancy greater than 17 images, anarea of at least one overlapping overview portion between a first andsecond overview image is greater than or equal to 50% of an area of oneof the first and second overview images, and an area of at least oneoverlapping portion of a first and second detail image is less than orequal to 10% of an area of one of the first and second detail images.23. The system of claim 22, wherein the second system includes between 5and 13 second image capturing devices that capture strip-shapedsub-portions of the overview images.
 24. The system of claim 22, whereinthe second system includes 3 sub-arrays of second image capturingdevices, wherein a first sub-array captures detail nadir view images, asecond sub-array captures first oblique detail view images, and a thirdsub-array captures second oblique detail view images.
 25. The system ofclaim 22, wherein a forward overlap of the overview images is between94% and 99% with overview images that are longitudinally adjacent to anoverview axis.
 26. The system of claim 22, wherein the second system isconfigured to capture images corresponding to multiple different views,including one or more oblique views and/or one or more nadir views. 27.The system of claim 22, further comprising: a processing component thatcompresses image data local to each image capturing device; and a solidstate data store local to the image capturing devices that is configuredto store image data in parallel with regard to each image capturingdevice.
 28. The system of claim 22, wherein one or both of the firstsystem or the second system are configured to include a removablemounting system such that the image capturing devices may beinterchanged with different image capturing devices.
 29. The system ofclaim 22, wherein a forward overlap of the overview images is between98% and 99% with images that are longitudinally adjacent to the overviewaxis.
 30. The image capturing system of claim 22, wherein the detailphotomaps have a ground-pixel resolution of 10 cm or less.
 31. A methodof capturing images comprising: capturing, via a first system thatincludes a plurality of first image capturing devices arranged on aplatform in different poses, overview images having an overlap to causea same ground point to be visible and measurable on at least 30 of theoverview images captured; capturing, via a second system that includes aplurality of second image capturing devices, detail images having ahigher resolution than corresponding portions of the overview images andrelated to each other along a detail axis; after the overview images andthe detail images are captured, performing a bundle adjustment processusing at least overview image overlaps and the detail images, todetermine spatial relationship information among the overview images andthe detail images, and to determine interior orientation and exteriororientation of the detail images based on the spatial relationshipinformation, wherein detail photomaps are generated from the detailimages using the interior and exterior orientation, and wherein theoverview images have a redundancy greater than 17 images, an area of atleast one overlapping overview portion between a first and secondoverview image is greater than or equal to 50% of an area of one of thefirst and second overview images, and an area of at least oneoverlapping portion of a first and second detail image is less than orequal to 10% of an area of one of the first and second detail images.32. The method of claim 31, wherein the detail images are captured witha resolution sufficient to produce a detail photomap having aground-pixel resolution of 10 cm or less.
 33. The method of claim 31,wherein the first image capturing devices and the second image capturingdevices are configured to capture respective overview and detail imageshaving overlap among the overview and detail images sufficient to enablethe bundle adjustment process.
 34. The method of claim 31, furthercomprising maintaining forward overlap between 94% and 99% in overviewimages that are longitudinally adjacent to an overview axis.
 35. Themethod of claim 31, further comprising capturing images corresponding tomultiple different views, including one or more oblique views and/or oneor more nadir views.
 36. The method of claim 35, wherein the one or moreoblique views and the one or more nadir views are capturedsimultaneously.
 37. The method of claim 31, further comprising:compressing image data local to each first and second image capturingdevice; and storing image data in parallel with regard to each first andsecond image capturing device via a solid state data store local to thefirst and second image capturing devices.
 38. The method of claim 31,further comprising arranging multiple first image capturing devices atdifferent poses to maximize an amount of ground feature observationscontained within multiple overview images taken by the first imagecapturing devices, maximization of the amount of ground featureobservations providing data sufficient to extract accuratephotogrammetric bundle adjustment solutions.